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Considerable research has gone into the development 
of student questionnaires for instructional evaluation. Somewhat less 
effort has been devoted to development of methods for diagnosis and 
remediation of instructional problems, but such methods are needed in 
any comprehensive instructional evaluation system. This paper will 
focus on diagnostic evaluation as a formative process, with 
discussion of the timing of evaluative procedures and the forms that 
';:hese procedures could take. The analogues of the statistical 
w!Oncepts of validity and reliability in diagnostic evaluation will be 
examined. Examples of techniques found useful by instructors and 
evaluators working in this area will be presented, (Author) 
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Much has been written about evaluation by students for the purpose 
of judging the quality of instrucfon. Less has been said about what to 
do if the students rate the inufcruction as less than satisfactory; yet 
"the Improvement of instruction" is a phrase commonly found in rationales 
for the development of evaluation questionnaires. To develop instruments 
which allow us to be critical of instruction without also developing 
structures and methods to help instructors improve is to leave half the 
job undone. This paper will focus on that neglected half. 

Improvement of Instruction 

There are several implications of the catch-phrase "the improve- 
ment of instruction". One implication is that instruction can be changed, 
a second is that change can lead to greater satisfaction for those 
involved, and a third possible Implication is that changes may lead to 
measureably better learning. 

We know it is notoriously difficult to change teaching habits. 
Orlasky and Smith (1972) have said that changes in methods of instruc- 
tion are apparently more difficult to make successfully than changes in 
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curriculum or administration* In the last decade the literature on 
higher education has expressed concern about the lack of properly 
trained college teachers and the inadequacies of most current doctoral 
programs in preparing future faculty members for their teaching roles* 
Meanwhile, the press for accountability continues to increase, and the 
need grows for more workable systems directed toward aiding faculty as 
they seek Instructional change. 

Hopefully, changed instruction means inqproved instruction which 
at the very least implles^ reater satisfaction f qr_^j^udents.,, instructors, 
or both. It also implies its antithesis; former dissatisfaction with 
instruction which can provide an impetus to change. The perception of 
.a '^problem" is always a necessary antecedent to change. If we take 
Instructional improvement to mean the alleviation of an individual 
instructor's perceived problem, then we have some basis for activities 
leading to diagnosis and remediation. 

Many would contend that "Improved instruction" can be claimed only 
when measureable increases in achievement occur at the same cost per 
student or when equal learning occurs with a lower cost per student. But 
as long as there is no universally acceptable criterion for quality teach- 
ing, let's leave the definition of improvement up to the instructors 
themselves* Often instructors are particularly concerned with student 
and teacher satisfaction* llhile such criteria* of good Instruction are 
less academically rigorous than many would prefer, they are not antithetical 
to more exacting definitions. Indeed, depending on the nature of the 
instructor's perceived problem, remediation may well Include attention to 
learning outcomes or the efficiency of instruction based on results. 

Stake's (1967) Countenance Matrix (Figure 1), originally conceived 
as a plan for the evaltiation of educational programs, can be a useful 
guide for people working cooperatively with instructors as they wrestle 
with problems arising in individual courses. The three rows of Stake's 
matrix, which call for logical contingencies between antecedent conditions, 
transactions, and outcomes, can be a powerful analytical tool to focus 
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attention on potential problem areas In Instruction. The emphasis on 
antecedents and transactions Is a welcome addition to the more usual 
focus on outcomes. 

The process used to provide Information and develop the credibility 
necessary to affect Instructor Judgments seems consistent with a "social* 
interaction" change model; one that sees change occurring as a result of 
a social relations network. Havelock (1969) reports substantial empirical 
evidence for this medel. Diagnosis and remediation of perceived Insjruc** 

tlonal difficulties Is most likely to occur given a method of analysis 

which Is comprehensive as well as credible and which operates In a system 
conducive to encouraging changes of judgment. 

The poor Instruction which many students report may be partly due 
to the instructor's lack of knowledge about factors In teaching and 
learning. Because most university professors have not been formally 
trained to teach, it is questionable whether or not they can be held 
accountable for this shortcoming. Given the increasing press for account-* 
ability 9 with no end yet in sight, it is important for those of us 
interested in the evaluation of instruction to give searching considera- 
tion to evaluation methodologies and systems encouraging remediation 
which reach beyond those most commonly used today. Indeed, any evalua- 
tion plan which does not inc^lude an adjunct service which can effectively 
assist faculty members in the iiiq>rovement of their teaching performance 
is ethically questionable. 

In viewing the diagnosis and remediation of instructional problems, 
we will distinguish between three levels of instructional evaluation, 
each level directed toward fulfilling a particular function. What we 
call Level I evaluation fulfills the need for general, comparative, sum- 
native information. Because comparisons are made across a wide variety 
of course types, the questions on which Level I evaluation must be based 
are of such a general nature that they are of little use, except in the 
broadest sense of the term, to the diagnostic. process. The second level 
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of evaluation Involves gathering Information about more specific 
attributes of classroom activities. Instruments useful for this kind 
of evaluation can be developed from sets of questions aimed at particular 
aspects of instruction. This level of evaluation is partially diagnostic 
in that it can help identify particular areas of success and failure in 
a given course. To the extent that Level II questionnaires consist of 
questions designed to apply to a variety of instructional settings » they 
can be standardized. Most student questionnaires in use today seem to 
be designed mainly to gather Level information. 

If the results of Level I and Level II evaluations lead the 
instructor to believe that there is an instructional "problem*'. Level III 
(diagnostic) evaluation is called for. This level of evaluation is course 
specific and almost always initiated by the instructor. Level III prob- 
ably has Its greatest payoff if used fairly early in a course, and if 
its focus is generated by a previous Level II evaluation. The aim of 
diagnostic evaluation is to help in remediation of aspects of Instruc* 
tion thought to be problems. Therefore, the exact format of the evalua- 
tion must be dictated by the nature of the problem. 

Level III evaluation often produces information in a form which 
defies statistical treatment. For instance, the usual statistical 
approach to questions of reliability can rarely be used. However, it is 
possible to look for basic consistency of opinion among multiple raters. 
Such consistency is often surprisingly easy to obtain in cases where a 
problem is severe, especially if Instructions given to the raters have 
been carefully formulated. A related approach to the question of relia- 
bility involves use of a two-stage evaluative process. After an initial 
(Level II) evaluation has tentatively identified some specific problems 
which seem to be present, a further evaluation in which the evaluators 
are asked to focus and comment specifically on these possible problems 
can be conducted. The criterion of consistency would then be applied to 
the latter evaluation. 
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The word "validity'' can mean several different things In the 
context of Level III evaluation. One can look for the equivalent of 
criterion-related validity by measuring whether change results in 
instructional improvement which can be objectively measured; in the 
long**term view such validity is important. However » in the formative 
stages of diagnostic evaluation the most inportant type of validity is 
the credibility given to the Information by the Instructor. There is 
very little chance that his instruction will change unless he perceives 
the evaluation information to be meaningful and appropriate. In this 
second sense, validity can not be objectively measured » but nevertheless » 
it must be present. 

While reliability and validity are liq>ortant evaluation concepts, 
the aim of diagnostic (Level III) evaluation Is specificity, not generaliz 
ability; and the validity of the enterprise rests on the instructor's own 
criterion of course Improvment. 

POTENTIALLY USEPDL TECHNIQUES 
Open^-Ended Questions 

At present the most common way of atteiq>tlng to gather diagnostic 
information is through the use of open-ended questions administered to 
students separately or in conjunction with a forced-choice questionnaire. 
Before an open-ended questionnaire can be considered to be diagnostic, 
questions must he focused sufficiently to elicit analytical comments 
from students. This is particularly important because students commonly 
respond in terms of adjectives about the course, but seldom explain their 
reasons for choosing particular adjectives. Directions to the student 
which clearly explain the nature of the Information that is being sought 
can help overcome this tendency. Information gained from the more typical 
Level II (forced-choice) instruments is invaluable in guiding the develop- 
ment of open-ended questions suitable for diagnostic purposes. Experience 
with open-ended questions has shown that even when students rate a course 
quite highly on forced-choice instruments, they can be helpfully critical 



If they understand the uses to be made of their Information » and if 
questions are sufficiently focused* 

An Increase In student response sensitivity can be gained by 
ordering questions logically on an evaluation form. For instance, an 
open-ended item requesting an analysis of an instructor's lecturing 
technique could immediately follow a forced-choice item which asks the 
student to rate the lecturer on a four or five-point scale. Such 
ordering discourages students from making general good'^bad statements 
in response to the open-ended item* The Level 11, forced-choice ^ infor- 
mation should indicate the overall opinion quite well, while the 
Level III, open-*ended, information is likely to provide helpful insight 
into the reasoning behind the forced-choice response* 

Once systematic data has been gathered, the potential for change 
has been established. That change may come through a process of self- 
evaluation during which an instructor weighs the credibility of the 
information and ponders his own reaction* Often the instructor will want 
more information to consider and will seek additional avenues to attain 
it* At the University of Illinois, Urbana-Champaign campus, the desire 
for additional information sometimes leads an instructor to contact a 
member of the Office of Instructional Resources* One of the purposes of 
establishing the Office of Instructional Resources was to improve the 
quality of undergraduate instruction, and a number of people are available 
to work with individual instructors for that purpose* Over the last five 
years a variety of techniques have been tried in our attempt to gather 
penetrating data and conduct activities which result in changes in 
instruction* 

Interviewing 

The exact activity recommended depends on the instructor's percep- 
tion of the problem, and on the types of solution which he can accept as 
valid* For instance, when the instructor does not believe that open-ended 
questions have provided sufficient Information, it is possible to arrange 
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to Interview students In more depth. In some Instances » again depend- 
ing on the predilections of the Instructor, six or eight students from 
the class have met with the Instructor In an Informal setting to discuss 
problems in more detail. In cases In which the Instructor would not be 
comfortable In such a situation » someone from the Office of Instructional 
Resources may sit in his place and act as an intermediary. Our exper- 
ience has shown students to be quite willing » even anxious » to be critical 
and helpful in such settings. 

In other instances » a random sample of students in a particular 
Instructor's class have been interviewed Individually. Experience gained 
from this activity again suggests that students are most willing to 
provide Information that is penetrating and useful. The written reports 
based on student interviews may be passed along to the instructor without 
comment ^ or more commonly, can be discussed with the Instructor • 

TV Tapes 

Videotaping a class In progress can be a frightening and yet 
enlightening diagnostic tool. Because instructors are often defensive 
about having TV equipment brought into a classroom. It Is an activity 
which demands instructor confid.ence and acceptance before it is used. 
At the Office of Instructional Resources we make arrangements for instruc- 
tors to view the tapes by themselves If they choose, and in many cases 
they will do so but will later request that other people view with them. 
It is seldom necessary to view the whole tape. Instead, watching just 
some portion of it usually leads into a discussion of problems made 
apparent by the tape. 

Remediation Possibilities 

In some situations students may be helpful in specifying the exact 
nature of an instructional problem, yet not at all helpful in suggesting 
possible solutions. In such situations a specialist from the Office of 
Instructional Resources, with knowledge both of the learning process and 
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of instructional methods » is frequently in an advantageous position to 
help an instructor broaden his outlook* It is at this point that sug-* 
gestions to try different approaches to instruction may fall on receptive 
ears. If the problem lies in the area of testing and grading, instructors 
can be shown ways to improve their test items and encouraged to develop 
banks of hig|h quality items. Other types of problems point to greater 
use of graphics in a course » increased attention to course assignments, 
or iiq>roved content organization* Still others suggest a wholly different 
approach to teaching » such as the audio-tutorial method offers* The pro«* 
cess of ''brain-storming" using students and/or teaching assistants along 
with the Instructor might be fruitful. 

Joint Appointments 

Educationalist!?., curriculum developers » and test and measurement 
experts occasionally lack credibility to physicists, botanists , geogra- 
phers, etc* One vay of alleviating this problem is to establish Joint 
staff appointments between the local Office of Instructional Resources 
and academic departments. The persons filling such appointments are 
subject matter specialists who, in addition, have a strong interest in 
instruction. Because such persons come equipped trith a built-in know- 
ledge of the language and orientations of their field, they can establish 
a degree of credibility difficult for those from other disciplines to 
obtain. While working closely with an Office of Instructional Resources * 
staff member, they can initiate appropriate diagnostic activities among 
their colleagues* A member of our staff, interacting with several such 
appointees, can have a much more extensive impact than would be possible 
when working on a strictly one-to-one basis with instructors. Joint 
appointments, therefore, are especially valuable. 

CONCLUSION 

We have previously mentioned the difficulties inherent in changing 
teaching habits. Because teaching methods and style are so intertwined 
with the personality of the instructor, and because personality changes 
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are not likely to be wrought » many think that instructional changes 
are not likely. Our experience in the Office of Instructional 
Resources suggests that there are many relatively unexplored activities 
that can lead to instructional improvement. If faculty is to be held 
more accountable for their instruction , we must be committed to develop** 
Ing more fully the activities which can lead to change. 
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